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Abstract 

No matter the care and caution employed in Space Station activities, accidents will happen. To prepare 
for this problem, NASA is constructing a robot, the EVA Retriever (or EVAR), whose purpose is to 
retrieve astronauts and tools that float free of the Space Station. Advanced Decision Systems is at the 
beginning of a two-year project to develop research software capable of guiding EVAR through the 
retrieval process. This involves addressing problems in machine vision, dexterous manipulation, real time 
construction of programs via speech input, and reactive execution of plans despite the mishaps and 
unexpected conditions that arise in uncontrolled domains. 

This paper concerns the problem analysis phase of our work. We use a walk through of an EVAR 
scenario to elucidate major domain and technical problems, and we conclude with an overview of our 
technical approach to prototyping an EVAR system. 


1. The EVAR Task 

In the event of an accident on the Space Station, NASA has decided to send a robot rescue vehicle in lieu 
of risking yet another astronaut in untethered, remote operations. The extra vehicular activity retriever 
(EVAR) is required to locate, rendezvous and return with an astronaut within 120 minutes, which is the 
duration of normal oxygen reserves. Calculations using an empirically observed maximum separation rate 
of 3.5 ft/sec and the known acceleration capability of the MMU show that the astronaut will at most be 5 
km away. 

Retrieval will be an interactive process, with the potential for teleoperated control of EVAR (moving the 
arms during grappling), autonomous behavior (search, navigation and spin cancelling), and user 
instruction ("extend your right arm while opening your hand"). As the scenario already assumes there 
has been an accident in space, there is little control over EVAR’s operating environment. The robot will 
require a great deal of flexibility. Since the situation is also life threatening, EVAR must be capable of 
operating even if some of its systems are nonfunctional, and if external instrumentation has failed. 

The basic scenario for retrieval of a free floating object has 7 parts: 

1. Activation 

2. Acquisition 

3. Rendezvous 

4. Grappling 

5. Return 

6. Transfer 

7. Deactivation 

In the nominal scenario, these parts occur in sequence, although there are many patterns for flow of 
control (for example, rendezvous and grappling can repeat if, say, a toolbox spills). 

We will discuss acquisition, rendezvous and grappling in some detail. A more complete discussion of the 
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EVAR scenario can be found in [4], 


2. EVAR Hardware and System Capabilities 

EVAR is a man sized manipulator platform seated in the Manned Maneuvering Unit, which provides 
mobility. The current design is anthropomorphic, with 2 arms and anon-specified hand/tool attachment. 
It carries a laser range finder (with 128 x 128 resolution). Potential sensors include: 

• A monocular TV camera (on the wrist and/or fixed to the chassis), 

• a radar, 

• a proximity sensor with spherical coverage, 

• a star tracker (low probability), 

• a Global Positioning System receiver (good to ~10 meters), 

• gyros for detecting own acceleration and for computing velocity and position, 

• a radio receiver. 

EVAR will be capable of accepting voice commands. 

The current processor configuration contains 4 transputers (currently 15 Mhz, T414 chips, to be 
augmented with five, 20Mhz, T800 chips). The current programming language is Occam, which will be 
changed to C. The total weight of EVAR (including the MMU) is 600-10001bs. Note fuel is highly 
constrained; total delta-v = 66 fps at system weight less than 600 pounds, or 1558 lb-sec usable impulse. 
Acceleration is 0.3 +/- 0.06 fps 2 for translation, and 10.0 +/- 4.0 deg/sec 2 rotation. 

The manipulator system will have both force feedback and proprioception (the ability to directly sense 
the angle of arms and joints). 


3. The Space Station Environment 

The Space Station has many of the aspects of a construction zone, whose physical composition will evolve 
through time. With respect to navigation, this means the immediate environment is cluttered, although 
three dimensional maps for completed portions of the structure will be available. At larger distances, 
there are no obstacles to movement, but it becomes apparent that orbits are accelerating reference frames. 
This affects trajectory calculations. From the point of view of computer vision, the environment is quite 
restricted; lighting is stark (numbers of point sources, no diffuse lighting), and the objects are metallic, 
geometric, and well modeled. However, it may be day or night, and the earth or the sun may necessarily 
be in the image. It is possible to instrument the Space Station, astronauts and equipment to a reasonable 
degree for the purposes of retrieval. 


4. Acquisition 

The goal of acquisition is to positively identify the location of the escaped object; specifically, its 
distance, speed, and direction of travel. This task subdivides into detection (identifying the object’s 
direction in space from EVAR), and measurement (identifying its velocity vector). Acquisition may be 
accomplished through a mixture of EVAR and non-EVAR resources. For example; user input, Space 
Station radar (if present), the EVAR camera, or laser range finder. Acquisition may also produce partial 
results; for example, a second astronaut may be able to identify the object’s direction, but not its 
separation velocity. 

There are several basic strategies for performing acquisition: 

1. Scan for the lost object in place. 
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2. Move to an observation location and scan for the lost object. 

3. Physically conduct a search pattern if scanning is not sufficient. 

4. Move in an externally provided direction and scan for the object while moving. 

EVAR will need to select among these options, with possible user interaction. 

It is clear that instrumentation for both detection and measurement exists, but the actual selection 
becomes an issue when space station safety, mass, and environmental constraints are taken into account. 
In particular, current design does not call for radar (owing to its mass and interference with EM sensitive 
instruments), while backup instrumentation is also desired. 

The most obvious options for instrumentation are as follows (D indicates ability to perform detection, M 
is for measurement): 

Passive sensing: 

• optical, by EVAR or tethered astronaut D 

• triangulation of astronaut- carried beacon D,M 

• interferometry, based on a beacon D,M 

• motion detection against a star background D, with doppler from a beacon D,M 

Note that passive detection will not work on all objects (small dark tools are generally hard to see, 
although they can be found with a plausible motion detector). Some form of prearranged beacon is 
required to support measurement with passive sensors. 

There are two obvious active sensing techniques: 

• radar (D -f M)? 

• laser range finder D 

It is not clear that a single radar can perform both detection and measurement within Space Station 
power, time, and mass constraints. In the absence of radar, a motion detector with EVAR’s laser range 
finder can perform acquisition, or we can employ a space station based strobe for optical acquisition 
(which has poor positional accuracy), with triangulation on the returned signal. Note that these backups 
may be less capable or less tolerant of environmental conditions than an appropriate radar. 

5. Rendezvous 

The rendezvous process involves trajectory calculation and path execution (in 3D), and culminates with a 
standoff maneuver (position holding action) next to the object to be rescued. The rendezvous phase may 
also involve completion of the detection and measurement tasks initiated during acquisition. EVAR must 
obtain a positive visual fix on the object during rendezvous. 

Clohessy-Wiltshire equations provide the appropriate trajectory calculations for objects in orbit, unless 
the parameters of motion are incompletely known. Time efficient, fuel efficient, and "maximum 
probability of acquisition" trajectory calculations are all relevant tools. A technique is required for 
measuring and maintaining a constant distance from an irregular, rotating object. 

The path execution function involves travel in the potentially cluttered environment near the space 
station, with avoidance of moving obstacles. 
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6. Grappling 

Grappling begins when EVAR is close to the object, and stationary with respect to its center of mass. It 
ends with EVAR physically coupled to the object such that it can be towed back to the Space Station. 
The problems of grappling come from the fact that the object may be tumbling in space; its size, shape, 
the complexity of its movement, and its rotational energy all affect the grappling techniques and tools 
that can be safely applied. Once the physical connection is made, conservation of angular momentum 
dictates that EVAR will inherit some unknown spin (in 3 axes) about the EVAR-object center of mass. 
This spin should be cancelled, and the object tethered to EVAR before it returns to the space station. ' 

The need to grapple with a tumbling object raises a few interesting questions. Can EVAR exploit 
properties of motion in free space to simplify the mechanics of interaction? What kinds of grappling tools 
can be/should be employed? What role does the astronaut play in the grappling process? Is he/she the 
object, or the director? 


6.1. Properties of motion in free space 

Unfortunately, in all but the simplest situations, there is no easy way to simplify the dynamics of 
interaction with a free-floating object. Exceptions are objects which don’t rotate, or which spin perfectly 
about one axis, as in the case of a normal communications satellite. In precession, motion is still about 
one internal axis, but that axis moves in space. As a result, EVAR cannot adopt the strategy of matching 
spin along some fixed axis, and leisurely grapple with the object, achieving an apparent 0 velocity 
encounter. 

In more detail, every object has 3 natural, or principal axes, one with maximum, one with minimum, and 
one with an intermediate value of rotational inertia. In space, a rigid object with no applied torques can 
have constant spin about the maximum or minimum axis. Spin about the intermediate axis is metastable 
(recall the high school experiment with a spinning tennis racket - there are certain spins it will not 
maintain). 

The motion of non-ideal objects (astronauts) also evolves. A pure rotation in the presence of even minor 
perturbations becomes a tumble (motion about more than one axis). A tumble in the presence of 
dissipative forces becomes a pure rotation, but only after a substantial period of time; days, not minutes 
as would be required to aid retrieval. 

The motion of the instantaneous axis of rotation of a tumbling object is constrained about its angular 
momentum vector (although not in an easily observable way). This suggests EVAR can simplify (to what 
degree?) the object’s apparent motion by diagnosing and then aligning itself with the object’s angular 
momentum vector. Intuitively, this will also simplify the net interaction; if EVAR and the object have 
parallel angular momentum vectors, grappling will exert no net torque, and the EVAR-object system will 
have the same orientation after all spin is cancelled. 


6.2. Direct Coupling 

In direct, or manual grappling, EVAR simply uses its manipulators to establish an immediate physical 
connection with the object. This exposes EVAR to impact, and will result in sudden momentum 
exchange, so the technique makes most sense for objects with small amounts of rotational energy. 

The task of directly grasping a tumbling object is by no means trivial, and in many ways beyond the 
current state of the art. For EVAR to use its hands, it will need to model the object’s shape (a scene 
analysis problem with dynamics), select a location to grab, diagnose its tumble (mathematically plausible 


228 



for rigid objects, but sensor intensive), plan a path to the appropriate rendezvous (in time and space), and 
execute the grasp with some form of force feedback control. If it is a rotating wrench, only EVAR’s limb 
needs to move. If it is a slowly rotating astronaut, the grasping action may involve EVAR translation, 
rotation, and use of both arms (and many joints) in a coordinated, and compliant two handed grasp. 
Some object recognition capability is also implicated; it better to grasp an astronaut by a leg than by the 
head, and some equipment (such as radio antennae) are too fragile to use as handholds. 

There are several ways to simplify this task. One is to ignore object rotation by grasping quickly. This 
will be difficult for both teleoperation and autonomous control, especially when it involves complex hand 
or limb motions. A reasonable solution is to instill manipulator reflexes to make a set of jaws close 
quickly on physical contact or on signal. 

The second solution is to employ grappling tools which can adapt to various combinations of object mass, 
movement, and shape. For example, if EYAR employed a simple butterfly net, it could entirely avoid the 
recognition, modelling, and dexterous manipulation tasks discussed above. 


6 , 8 . Loose Coupling 

The objectives of loose coupling are to spread the EVAR-object momentum exchange over time, and to 
avoid dangerous interactions between EVAR and moving objects. This requires specially designed 
grappling tools. The ideal tool is insensitive to all of the object parameters (shape, identity and motion) 
discussed in the previous pages. 


A representative list of tools is shown below. (This list is a compendium of suggestions spanning ideas 
from silly to clever, and buildable to completely impractical.) Several are illustrated in figure 6-1. 

1. A rope with a lifesaver, deployed for the astronaut to grab. 

2. A rod and reel apparatus, with a rotating end attachment to allow motion. 

3. A powerful adhesive on the end of a stiff wire probe. 

4. A contact activated clamp attached to the end of a manipulator or wire (as in 2 above). 

5. A butterfly net. 

6. A pellet gun carried by the astronaut, used to reduce spin. 

7. An electrostatic aligner, with sprays for inducing a dipole field on the object. 

8. A bolo, used to snare the target. In theory, the weights at the ends of the bolo spread out, 
causing the rotational inertia of the object to increase and its rotational velocity to decrease, 
simplifying further grappling. 

9. A compressed gas apparatus, deployed onto the astronaut, mechanically designed to align the 
thrust opposite object rotation. 

10. A modified TOW missile, guided remotely from the space station (an EVAR backup). 

11. Rotating nets embedded in a pair of clappers. 

12. The Brupiro Grappler: an annular ring containing movable, rotating snares, manipulated by 
an external, swivelling handle. This device has 3 degrees of freedom, and permits capture 
with no immediate momentum exchange. 

13. A spherical shell with extensible arms that matches the tumble of the object at the instant of 
capture. 

14. A momentum leech. The leech extends telescoping arms with attached masses until the 
angular velocity of system is damped. It then drops off and retracts its arms (spinning up in 
the process), while EVAR grapples with the now slowly moving astronaut. 


Many of these tools have significant mechanical problems. The lifesaver is practical and inexpensive, but 
the astronaut could grab EVAR just as easily. With ideas 2 and 3, a tumbling object will attempt to roll 
up the line and tangle itself in the process; this produces astronauts packaged in large balls of string. The 
pellet gun (6) has a nasty side effect; it sends a barrage of high velocity material into the environment. 
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There is also a small problem with choosing a direction to fire, since tumbles are notoriously disorienting. 
A bolo might slow the astronaut’s spin, until it reverts to its historical role as a weapon. No one has 
admitted to a design for the electrostatic aligner. The compressed gas jet (9) is mechanically complex, 
and will also apply an asymmetric torque which will produce odd motion (probably a spiral translation). 

The remaining ideas are actually quite serious, although they require more elaborate machinery. The 
clapper is under design at NASA. The Brupiro grappler is the invention of this author (and Dr. Bruce 
Sawhill). The tumble matching shell is under devlopment by Capt. Don Idle at the University of Texas at 
Austin. Before dismissing the leech, note that an astronaut with 50 kg-m of rotational inertia can be 
reduced to l/10th his initial angular velocity with five, 1kg weights at the end of 10 meter poles. 

6.4. User Interaction During Grappling 

If we assume that the lost astronaut is not terribly disoriented, he/she will almost certainly insist on 
being active throughout EVA retrieval. This suggests some form of a vocabulary for verbally directing 
EVAR. 

A simple < action > <object> < direction > < magnitude > grammar is an obvious (though primitive) 
start; define nouns for pieces of robot anatomy (wrist, elbow, shoulder, arm, hand, body), verbs for 
actions (move, open, close, flex, rotate), and keywords for direction, magnitude and speed (forward, 
backward, clockwise, 20 degrees, fast, slow). Use these terms to form robot commands; "move body 
forward slowly", "extend left arm", "flex right elbow 30 degrees", "rotate wrist clockwise", "close 
hand". 

We might also introduce keywords for temporal coordination and sensory conditions such as "while", 
"then", "on contact", etc. This would allow expressions such as, "extend both arms while opening both 
hands", which is useful as a method of delivering an object, or, "on contact close hands quickly", which is 
a method for grasping a rotating object. 

This vocabulary essentially defines a programming language for controlling simple robot actions. On the 
input side, the voice recognition and natural language understanding tasks appear almost ideal candidates 
for automation; the vocabulary is limited and sentence meanings correspond to robot actions. The 
program generation task may be more difficult, depending upon the underlying robot action primitives 
and the complexity of the control structure the user desires. Michalowski [2] has addressed this problem 
in the design of robot arms and wheelchairs for assisting the severely handicapped. He has produced 
programmable, operational robots, but without reactive control. 

Controlling detailed manipulations by voice command is likely to be quite tedious, and too inefficient for 
use in retrieval scenarios. A solution is to raise the level of discourse, allowing the astronaut to interact 
with EVAR in terms of a vocabulary of physical behaviors. For example, the astronaut might say, "use 
the back-in procedure", or "use the clapper procedure", which causes the robot to begin execution of a 
predefined multistep plan. The behaviors function as contexts which provide specific command options to 
the astronaut in addition to causing robot action. So, for example, the "back- in" procedure directs 
EVAR to rotate 180 degrees, apply gentle thrust towards the astronaut, and terminate when it senses 
impact. Commands for controlling EVAR speed, and for initiating spin cancelling become available in 
this context. (Behaviors can be used as the basis for segmenting EVAR’s sensing, processing, and resource 
requirements. See section 7.) 
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6.5. Return 

Planning for the return trip involves acquisition and trajectory calculations as before, but in this case it is 
reasonable to assume EVAR knows its velocity and position relative to the space station. 

The act of moving towards the space station with the object in tow is a different proposition. In this 
situation, the JEJVAR/object total mass and center of mass are both unknown , and can be significantly 
different from EVAR values alone. This suggests an adaptive control solution, where the control 
parameter is units of thrust, and the measure is units of deviation from the desired (or expected) direction 
of travel. (Measurement of thrust, calibrated in Newtons, is not required.) 

If exact numeric solutions are desired (for example, to know how much fuel is going to be available or if 
the task can in theory be solved) it may be possible to run a simple experiment with EVAR, using MMU 
gyros to measure acceleration against a known application of force. The proper instrumentation for this 
task is currently not present on the MMU; it lacks calibrated accelerometers for measuring translation 
and rotation rates, and a force sensor for measuring the moment actually being applied. 


7. Scenario analysis 

This scenario brings out a number of technical problems in machine vision, dexterous manipulation, man- 
machine interaction and robot programming. 

Concerning machine vision, the environment is both well known and restricted. However, the image 
understanding problems are non-trivial. A representative problem is; extract an astronaut from a scene 
with the sun in the background (using video or laser range data), and autonomously identify his leg, vs. 
his head. Track this object, and develop a predictive model for its location. Note that the astronaut will 
be tumbling, and moving his limbs. This problem clearly requires research and application of the 
technologies of optic flow, model based vision, and shape detection. 

A second obvious problem concerns navigation in 3 dimensions with avoidance of moving obstacles. No 
such vision/planning systems exist to date; the closest examples are in autonomous land vehicles 
(although land motion is arguably harder). 

The manual grappling task brings out issues in dexterous manipulation. Here, there is a need for rapid 
planning and execution of motion (to grasp a tumbling wrench while a predictive motion model is in 
effect), with a potential need for compliant response and coordinated action (use of two hands, vs. one). 

Concerning man-machine interaction, the discussion of grappling with an astronaut brought out a need 
for real time construction of small programs via speech input, or for an extended EVAR command 
vocabulary. 

Finally, the uncontrolled nature of the scenario as a whole indicates a need for a different approach to 
robot programming; EVAR cannot view retrieval as execution of a preset plan. To illustrate this point, 
consider a few things which can "go wrong" during retrieval: 

1. The object may be an astronaut, tangled in a girder, rotating at high speed. As a whole, that 
system will be outside the tolerance of the available grappling tools, even though the astronaut 
alone is within EVAR abilities. 

2. Both EVA astronauts will be lost, EVAR will bring one to the other, and end up functioning 
as an assistant during grappling instead of the primary active agent. 

3. EVAR will encounter mechanical problems. Not all sensing systems will be operational. 
Communications will temporarily fail. Tools will malfunction (for example, the momentum 
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leech won’t disengage, or the clappers will not deploy completely). Beacons will not work. 

4. Initial grappling will fail, and make the situation worse. (EVAR will experience a collision 
and inherit significant momentum.) 

5. EVAR will lose orientation during spin cancelling, and will have to reacquire the Space 
Station. Something unknown will go wrong and it will have to be manually directed 
(teleoperated) back to the Space Station. 

6. There will be critical time pressures during grappling. 

7. The object will obscure EVAR sensors after grappling, or, it will get tangled in the tethering 
process. 

8. The pod bay doors will not open. 


The salient feature of these examples is that unexpected complications will stretch preparations for each 
of acquisition, rendezvous, grappling, return and transfer. Substantial improvisation may be required. 
This argues for a view of EVAR as toolbox that provides a range of applicable behaviors, as opposed to a 
single purpose, preprogrammed device. 


8. Technical Approach 

Advanced Decision Systems is at the beginning of a two-year project to develop research software capable 
of guiding EVAR through the retrieval process. In order to build a functional system in that time frame 
(or in any reasonable future period), we believe that the technical problems discussed above cannot be 
tackled head on. Our approach relies on the following key ideas; 

• We simplify vision processing by use of user assisted scene interpretation (after Lawton [1]). 

• We avoid manipulation tasks through use of grappling tools. 

• We support user interaction by programming EVAR in terms of a vocabulary of physical 
behaviors. 

• We provide reactive response by embedding these behaviors in an architecture which examines 
all possible actions each time step. 

The goal in user assisted scene interpretation is to map recognition and modelling problems into tracking. 
(For example, the user identifies the astronaut’s leg in the image, selects a generalized cylinder from the 
model library and provides an initial fit to the rotating object. The system takes over the tracking task 
from this point.) 

The last two points above address a major issue of current robotic control; the need to have plans but 
also react to external events as they occur. A system architecture for reactive plan execution is shown in 
figure 8-1. 


The main feature of this architecture is the abstraction called the current program , which is a constantly 
modified data structure that maintains all the behaviors, tasks, and situation triggers on mind of EVAR 
at any given time. 


Computationally, each behavior is a strong context for specifying action. Examples are to "maintain an 
object fix", to "maintain distance", "maintain orientation", or at a larger scale, a "rendezvous mode" 
which can be hierarchically decomposed into simpler contexts. Each behavior requires limited world 
knowledge, supports specialized sensor processing routines, and has triggers which respond to events 
natural in that context (e.g., impact during grappling). For the purposes of arbitration, each behavior 
can also identify the resources it requires. 

The use of strong contexts allows a unique approach to plan representation; after Schoppers [3], each 
behavior encodes all paths from the current situation to the goal. This provides a great deal of reactivity. 
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Figure 8-1: An Architecture for Reactive Plan Execution 

In particular, the robot is not dependent on previous actions working as desired; the step relevant to the 
current situation is always applied. Longer plans, such as the high level EVAR retrieval sequence, are 
expressed as sequences of contexts together with their transition criteria (e.g., rendezvous transitions to 
grappling when the standoff maneuver is in process). 

The basic decision loop of this architecture is as follows: 

1. determine if any context switch criteria have been met, if so, activate and deactivate the 
appropriate behaviors, 

2. process all behaviors in the current program one cycle (this involves both diagnosis of the 
current situation and selection of the appropriate action) 

In this view, a plan defines appropriate decision contexts, and is treated as a set of suggestions, to be 
taken as the immediate situation allows. 
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